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ABSTRACT 


A hypertext is a new manuscript form that consists of a 
computer-based collection of related documents coupled with all 
their cross-references, footnotes, Annotations and other 
associations- Using a television-like computer terminal, a 
student, instructor, or researcher can conveniently access this 
collection of documents. The user can read the material, pursue 
cross-references, retrieve related documents, take notes, and add 
references and other material. 

The purpose of the experiment reported here was to determine 
whether significant educational benefits could be obtained 
through the use of a hypertext system as a supplement to normal 
class lectures and discussion. The vehicle in this case was a 
section of Brown University's English 16, "Critical Analysis of 
English and American Literature: Poetry." The results of the 
experiment, while statistically not definitive because of the 
small number of students involved, are extremely encouraging with 
respect to both quantity and quality of work produced. Some of 
the observations and benefits obtained may be summarized as 
follows: 

• The instructors all felt that the students made significant 
progress that can be attributed to the use of the hypertext 
system. Major improvements were in the areas of increased 
self-confidence and improved critical abilities. 

• The hypertext system was relatively simple to learn, whether 
or not the user had a technical background. 

• The hypertext group produced prodigious amounts of written 
work, despite the fact that English 16 is not primarily a 
writing course. 

• Independent evaluation showed students in the hypertext 

section to have improved more than students in the other 
sections. , 

• While active interaction with the material was not in the form 
that we expected, it was extensive and a great improvement 
over our earlier experiments. 

• The hypertext system encouraged efficient use of time and 
resulted in excellent per hour productivity. 
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• The benefits from classroom and hypertext experiences are 
complementary. Both are needed in the course format and 
neither should be emphasized too strongly at the expense of 
the other. 


In general, the results of the experiment indicate that the 
use of a hypertext system provides substantial educational 
benefits to students and instructors. In particular, the 
hypertext system provides a convenient and powerful medium for 
stimulating the communication of ideas among students and their 
instructors. The product of this environment is a significant 
new approach to the development of critical and writing skills. 
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Introduction 


1 I NTR ODU CTI ON 


A hypertext 1 is a new manuscript form that consists of a 
computer-based collection of related documents coupled with all 
their cross-references, footnotes, annotations and other 
associations. Using a tele vision- li ke computer terminal, a 
student, instructor, or researcher can conveniently access this 
collection of documents. The user can read the material, pursue 
cross-references, retrieve related documents, take notes, and add 
references and other material. 

The purpose of the experiment reported here was to determine 
whether significant educational benefits could be obtained 
through the use of a hypertext system as a supplement to normal 
class lectures and discussion. The vehicle in this case was a 
section of Brown University’s English 16, "Critical Analysis of 
English and American Literature: Poetry." The results of the 
experiment, while statistically not definitive because of the 
small number of students involved, are extremely encouraging with 
respect to both guantity and quality of work produced. 

This report is organized as follows: Section 2 describes 
the work done in preparation for the course. Section 3 contains 
a description of how the course was run, including the selection 
of student participants, the format of the course, and 
experimental controls. In Section 4 we present the results of 
the experiment. Section 5 contains concluding remarks and ideas 
for future work. Ihe appendices contain additional information 
which may be helpful in evaluating the experiment. Appendix A 
describes our efforts at disseminating the results of the 
experiment. Appendix B contains background information on 
hypertext and the development of the FRESS system. Statistics of 
interest are in Appendix C. Cost figures are in Appendix D. 


1 Theodor Nelson, "Getting It Out of Our System," in George 
Schecter (ed.). Inform ati on Retrieval : A Critical View, Thompson 
Book Co. (Washington: 1967), p.195. The word "hyper" is used in 

the sense of nan-licear.‘ 
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Preparation for the Experiment 


Equipment and Setting 
2 PREPA RATION FOB THE EXPERIM EN T 


2^1_§taff 


Prof. Andries van Dam - Principal Investigator 

Responsible for providing the overall direction of the 
e xperime nt . 

Prof. Robert Scholes 

Responsible, with the assistance of two teaching assistants, 
for organizing the poetry course, collecting materials for 
it and teaching it. 

Carol L. Chomsky - Director 

Responsible for coordination between technical and 
non-t echnica 1 aspects of the experiment, overseeing 

day-to-day operations, completing all technical work, and 
maintaining the hypertext computer system with the 
assistance of cne programmer. 

James Catano, Nancy Coraley 

Graduate students and teaching assistants responsible for 
helping to organize, gather materials for, and teach the 
poetry course. 

David Irvine 

Part-time programmer, responsible for helping with the 
technical development work. 

Several part-time typists 

Keyboarded the hypertext material. 

Drs. Donald Brown and Karl Zinn 

Consultants from the Center for Research on Learning and 
Teaching at the University of Michigan 
Richard J. Harrington 

Technical Consultant (Software) 

Martin J. Michel 

Technical Consultant (Hardware) 


2i2_EQUIPMENT_AND_S ETTING 


The equipment used for the experiment was an IMLAC PDS-1D 
display console and minicomputer connected by a 1200 baud (120 
characters per second) telephone line to Brown's central IBM 
360/67 computer. The IMLAC was located in a room devoted 
entirely to the English 16 hypertext section. The room also 
contained a desk, a large table, and several chairs to provide 
room for students to do other work while waiting to use the 
IMLAC. For the first 6 weeks of the course an instructor was 
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Preparation for the Experiment 


Technical Development 

always in the room to provide any necessary assistance in using 
the hypertext. 


2 i 3_TECHNICiJL_fRESS_DEJ2LOPMENJ_EOR_THE_EXPERIHEN2 

FRESS, the computer hypertext system used for the 
experiment, was developed under the direction of Andries van Dam 
at Brown University over a 7 year period. One year prior to the 
current experiment the project received funding, from the Exxon 
Education Foundation, which allowed development of many of the 
hypertext features used in both experiments. Further information 
on the FRESS system may be found in Appendix B. 

While the FRESS system was fully operational before the 
commencement of this experiment, several technical developments 
were required to allow more effective use of the system. Most of 
these developments were of a general nature so that they may be 
applied in future uses of FRESS. The work required approximately 
12 person-months. 

2. 3. 1 C o mmand Macros 

Command macros provide the facility for grouping several 
FRESS commands together under a new name. This feature allowed 
the instructors to create "commands" for the specific purposes 
needed by the students, thus simplifying the process of learning 
and using the computer system. For example, the instructors 
devised a command macro for inserting questions. The students 
could specify simply . iq and indicate where the the question 
should appear in the text. The macro would then cause a 
prompting message to appear on the screen for any additional 
information necessary. In this way the students did not have to 
learn the more complicated procedure actually invoked by the 
macro. In the previous Exxon experiment we believed that the 
hypertext was not frequently used because of the need to execute 
several commands; the existence of the new command macros was one 
of the factors that made this experiment more successful. 

The command macro feature was designed in November, 1974. 
The inplement ation continued from then through May, 1975. This 
involved writing a new routine to handle the command macros as 
well as modifications to many of the existing system routines. 

2. 3.2 D ispl ay Keywords 

Display keywords allow selective viewing of hypertext and 
sections of text. This development was necessary to allow the 
gradual introduction of material to the students that was desired 
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Preparation 

Course 


for the Experiment 
Format and Material 



by the instructors (see Section 3.2). It also allowed the 
students to control how much of the stu dent- genera ted text and 
hypertext they would see. For example, a student could, with a 
simple command, arrange to see only his or her own questions and 
comments but not those of other students. 

The display keyword facility was designed in February 1975. 
The facility was implemented in March and early April 1975. This 
involved alteration to existing routines to allow them to handle 
this new form of keyword. In addition we discovered that the 
keyword- handling routines had problems in handling complex 
retrieval commands. The month of July, 1 975 was spent correcting 
these problems. 

2. 3. 3 I M L AC C hanges 

The program running on the IMIAC minicomputer was changed 
substantially. In particular, the arrangement of material on the 
screen was altered to allow more text to be viewed at one time. 
A new set of special characters were drawn to include linguistic 
markings. An italic character set was drawn since many of the 
materials required a second character set. The changes to the 
IMLAC program were made in June, July, and August, 1 975. 

2. 3.4 Othe r Technical Wo rk 

Other technical work included minor enhancements to existing 
features and fixing bugs as they appeared. 



2 . 4 DETERMINATION OF COURSE FO RMAT AND COLL ECTI0N_0 F_CO UR SE 

MATERIAL 


Discussions were held in November and December, 1974 with 
the director and the instructors to determine more precisely the 
format of the course for those using hypertext. This included 
decisions on the types of material to be included in the 
hypertext and in what order and how they would be presented. The 
course was divided into ten units, each covering a principal poem 
and related material (see Section 3.2). The three instructors 
gathered the material _ for the units from various sources, 
including books, journal articles, anthologies of poems, and 
biographies. They culled from these sources those readings that 
best related to the central theme of the unit. Sufficient 
amounts and varieties of material were provided in order to give 
the students the opportunity to further select those materials 
most meaningful to their own perspectives. The collection of 
materials took approximately six person months of work by the 
instructors. They felt this was about four times the amount of 
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Course F 

ormat and Ma 

te 

ria 1 

time 

they would have 

spent 

in 

preparing for 

a course th 

ey 

had 

pre vi 

ously taught and 

twice 

the 

amount of ti 

me for prepa 

ra 

tion 

for a 

new course. Add 

i t i o na 

1 CO 

mments on inst 

ructor prepa 

ration 

ti me 

are in Section 4.4 

. 2. 







Materials for the 

first 

uni 

t were collec 

ted in Janua 

ry 

and 

the 

typists began ent 

er ing 

it 

at the end of 

that month. 

U 

nits 

wer e 

delivered to th 

e ty pi 

st s 

on the averag 

e of once a 

m 

onth 

af te r 

that. The firs 

t unit 

w as 

"structured" 

with hypert 

ex 

t in 

April 

, after display ke 

y words 

were implemented. 

Subsequent 

iy 

the 

uni ts 

were proofread 

and 

stru 

ctured as so 

on as they 


we re 

completely inputted. 

It took 695 person-hour 

s for all un 

it 

s to 

be keyboarded by the 

typists. 

It reguired 

approximate 

iy 

ten 

hours 

per unit tc d 

o the 

fi 

na 1 online p 

roof reading 

an 

d to 

str uc 

ture the unit. 

includ 

ing 

the creation 

of links 

to 

the 

alpha 

betical directory 

of poems 

. This task 

was shared 

by 

the 


instructors and the director. 

The two independent evaluators. Dr. Donald Brown and Dr. 
Karl Zinn, met with us in early April to discuss the way in which 
the experiment would be run in order to produce the most 
definitive results possible. We decided to use three sections of 
the poetry course as control sections and use a short exam at the 
beginning and end of the course as the basis of comparison. These 
controls are described in more detail in Section 3.3. 
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Course Mechanics 


Student Selection Process 


3_Q,0 UR MECHANICS 


3_. _1_STU DEN T_S EL EC 1I0N _PROCE SS 


We initially expected to be able to enroll 28-30 students in 
Prof. Scholes' section, slightly more than usual to allow us to 
break the section into control and experimental groups of equal 
size. Unfortunately, the enrollment was only 20. Of these, 15 
volunteered to participate in the experiment after a presentation 
by Prof. Scholes and an open house where the students had an 
opportunity to see a demonstration of the hypertext system. 
Twelve -were chosen to participate. The selection was made so as 
to allow the widest possible distribution of concentration areas 
and computer experience. After the initial selections and 
interviews with all participants by Dr. Brown, one of the 
students changed his mind and decided to return to the control 
section. He felt that he would miss class discussions too much. 
One of the three remaining volunteers was then chosen to take his 
place. 

It is important to note that none of the participants, 
including those with prior computer experience, took English 16 
because of the experimental nature of one of its sections. In 
fact, most of the students, again including those with computer 
experience, did not know of the existence of the experiment until 
the first day of class. The students' attitudes towards 
computers and the experiment ranged from enthusiastic to 
apprehensive but curious. 

Those who elect to take English 16, a non-reguired course, 
tend to be among the more verbal and articulate students at 
Brown. This was corroborated by the independent evaluations by 
Prof. Hirsch, as discussed in Section 4. 2. 3. 3. The composition 
of the class is summarized in the following table (the column 
headed with P indicates experience with poetry, almost 

exclusively at the high school level, and that with C, experience 
with computers) : 
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Course Mechanics 
Format of the Experiment 


Table 1 


Stu dent 




P 

c 

1 

M 

1 

Sociolcg y 

x 


2 

F 

4 

Applied Math. 


X 

3 

M 

4 

Biology 


X 

4 

F 

1 

American Civ. 

X 


5 

M 

1 

En glish 

X 


6 

F 

3 

Se miotics 



7 

F 

3 

Psychology 

X 

X 

8 

F 

1 

un decided 

X 


9 

M 

2 

Chemistry 


X 

10 

F 

1 

Education 

X 


1 1 

F 

1 

undecided 

X 


12 

M 

1 

Semiotics 

X 



3_;_2_FOR MAT_OF_THE_EX PERI M ENT 

The hypertext data base consisted of approximately 1000 
pages of material. This included two books and an assortment of 
materials that was divided into ten units. Each unit 
concentrated on one poem and included resource materials related 
to that poem. These central poems progressed in chronological 
order from "Sir Patrick Spans" (anonymous ancient ballads) 
through "Home Burial" (Robert Frost). Some of the other poems 
used were by Andrew Marvell, William Blake, W. B. Yeats, T. S. 
Eliot, and Wallace Stevens. 

The IBM 360 computer (and therefore the hypertext) was 
available approximately 115 hours per week. Students were 
scheduled for three 2-hour sessions per week; normally each unit 
of poetry was completed in these three sessions. This left more 
than 40 hours that could be used for the instructors to respond 
to the students, for extra sessions for the students, and for 
system maintenance (see Section 4.2.5). 

During the first session with a particular unit the student 
viewed only the central poem and then typed a response, from a 
paragraph to a page or ‘more in length. This response was linked 
to the poem so that other students could read it after they had 
made their own first responses. 

At the second session, the student again viewed the poem 
and, in addition, other primary source material which placed the 
poem "in context". This generally included biographical and 
historical information connected with the poem or poet, other 
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Course Mechanics 


Format of the Experiment 

poems by the poet being studied, and related poems by other 
poets. The student then typed a second response, sometimes 
prompted by specific questions entered by the instructor. These 
questions were available to be read at any time during the 
session. They might ask the student to reconsider his or her 
first response in view of the information presented in the second 
session (and in the first responses of other students) , or they 
might be of a more specific nature. For example, in the first 
unit's second session the students were asked to compare one 
ancient and one modern ballad, paying particular attention to 
similarities in techniques, structure, or themes and determining 
what made the modern ballad modern. In all cases, this second 
response was linked to the second session background information 
and was made available to other students who had completed their 
second session. 

On the third session the student was able to view a variety 
of critical materials in addition to all materials previously 
available. These articles of literary criticism were viewed last 
to enable students to form their own opinions about the poem 
first. After entering a third response, the student could view 
the third responses of the other students. 

Several units were structured a bit differently. In one the 
students studied the sonnet form, then wrote their own sonnets 
and finally critigued all the poems written by the students in 
the class. Another session was devoted to a study of the 
prepublication versions of a poem by W. B. Yeats, illustrated 
with excerpts from the poet's notebooks. 

The students controlled when they moved from one session to 
the next. The only restriction occurred when the students were 
expected to base their comments on the responses of all other 
students from a previous session. In this case each student was 
requested not to make a response until most other students had 
made their responses from the previous session. 

Shortly after the beginning of the course, the students were 
provided with notes files (online notebooks) in which they could 
record thoughts that would later be developed into their 
responses for a session. 

There was a final project in which each student was asked to 
discuss a particular t'heme or aspect of poetry, such as death, 
time, the car pe diem motif, the sea. They were requested to take 
notes in their own manuscript files and to make jumps (links) 
from there to relevant portions of the course material. Near the 
end of the semester they were to organize and edit these thoughts 
into a coherent discussion of the subject. 
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Experimental Controls 


The instructors used the computer terminal approximately 13 
hours a week. This time was used to determine the progress of 
the students, evaluate student responses, and answer questions or 
otherwise respond tc comments of the course participants. 


3. 3 EX PERIMENTAL CONTROLS 


English 16 is usually divided into three sections of 15 to 
20 students each. In this instance one section was further 
subdivided into two parts (1A and IB) to provide a total of three 
control sections and one experimental section. Students in the 
experimental section ( 1 A) used both a standard anthology and the 
hypertext manuscript as resource materials. The class met once 
or twice each week for 50 minutes each time. The instructors 
were Robert Scholes and two graduate student assistants, James 
Catano and Nancy Comley. 

The students in Section IB (also referred to as the "control 
section") had the same three instructors as the experimental 
section. They used the same standard anthology and had available 
to them on reserve in the library the material used in 
constructing the hypertext. This section met for three 50-minute 
classes per week. Sections 1A and IB covered substantially the 
same material although the methods of presentation were 
different. The students in Section IB were expected to keep 
personal "notebooks" in which they would write their questions, 
comments, and responses while they read at home. The notebooks 
contained their explications and reactions to the poems, written 
as they read at home; exercises from an introductory book on the 
analysis of poetry 2 ; some class notes; questions and other 
marginal notes; and responses to specific questions the 
instructors might ask. Most of the writing in the notebooks was 
devoted to responding directly to poems. Although the benefits 
gained from this procedure may prompt its use in the future, it 
was initially devised only because of the experiment, in order to 
mimic the hypertext section as much as possible. However, the 
amount of interaction available on the hypertext system could not 
be duplicated. Students and instructors in Section IB could not 
have available to them simultaneously all the work done by 
everyone involved in the course; this lack was significant. 

The other two sections were taught by instructors not 
directly involved in the experiment and the format was determined 


2 Geoffrey N. Leech, £ Ling uis ti c Guide to English Poetry (London: 
ho ng man , 196 9). 
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Course Mechanics 


Chronology 


by each instructor individually. All of the sections included 
many of the same poems. 

A common evaluation for all sections was devised to provide 
an independent comparison. On the first and last day of class a 
standardized test was administered to all four groups of 
students. The test consisted of the evaluation of a poem not 
included in the curriculum of any of the sections. The purpose 
of this exercise was to assess the progress of each student in 
his ability to respond tc a poem without the aid of resource 
materials related to that particular poem. 

In addition, the two sections taught by Prof. Scholes and 
his assistants were given identical final projects. 

Students in the experimental section were interviewed three 
times during the course by Dr. Donald Brown. Students in 
Sections 1A and IE answered a questionnaire which explored their 
view of the learning environment and their feelings about their 
relation to the material and the instructors. In addition, 
statistics were gathered throughout the semester on the computer 
usage by the students in the hypertext section. 


3. 4 CHRONOLOGY 


There were 67 class days (5 days per week minus holidays) 
during the semester, that is, 13.4 weeks. The 9 class days 
during the period September 16 through September 26 were devoted 
to explanation and demonstration of the system and selection and 
instruction of the participants. The first use of the computer 
for course work began on September 29. During the period 

September 29 through December 17 the students spent most of their 
time doing the three-session units on the poems as described 
above. After the Christmas break, during January 5-23, the 
students worked primarily on their final projects. The time 
after January 10 followed the end of the 67 class days and 
occurred during Brown's final examination period. 
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Results 


■4_R ESULTS 


The results discussed below are based on the following 
sources of information: director and investigator observations 
and notes, instructor observations and evaluations of students' 
work, the work itself, an independent evaluation of students' 
ability to critically discuss a poem, and comments and 
observations of the students. In addition, statistics were kept 
on computer usage by the hypertext group. In particular, these 
data included the date, time and cost of each student and 
instructor session with the computer, the quantity of work, the 
number of times each command was used, and the number of hours 
that the computer was not available because of equipment failure 
or other difficulty. At the end of the course, this data was 
sorted and aaalyzed. 

As mentioned earlier, two independent evaluators from the 
University of Michigan were involved in the project. Their work 
was supported by funds that remained from the earlier Exxon 
experiment. Their final evaluation will therefore be made to the 
Exxon Education Foundation. 

In this section we discuss what this varied collection of 
information shows us. It is, however, appropriate to point out 
at the outset that we can gain only a limited amount of 
information from the objective data. Though we found the 
information supportive of the value of a hypertext system in an 
educational setting, it would be improper to claim too much on 
the basis of the data alone. The induction base is simply too 
sma 11 . 

Thus, what seems to be most important in an experiment of 
this scope is the personal perceptions by the users about the 
effect of the hypertext system. Here we must lock to the reports 
of the students and instructors. The students were very 
enthusiastic about the course - ten of them rating it the best or 
one of the best courses they had taken. At least half of them 
attributed this and other benefits they found in the course to 
the hypertext system. * As we discuss below, in some cases they 
seem to be unaware of the source of the benefits. The 

instructors, especially Professor Scholes, had little doubt about 
why the course went well: 

When we planned this course we had certain hopes, 
things that we had hoped the course would accomplish 
based on our experience with ether poetry courses, but 
we didn't know whether any of them would actually work. 
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Computer Facilities and Software 


Now that it's over I think I can say that it worked 
beyond our wildest expectations. ...[ The students] 
communicated much more with one another, both through 
the hypertext and outside of the whole framework of the 
class. I think that the communication that they had 
with one another on the system, commenting on one 
another's work, reading one another's work, 
appreciating it, admiring it, criticizing it sometimes, 
generated a kind of rapport among the members of the 
group that is ver_y unusual. I can think of no way that 
can be duplicated without a system of this kind. 3 

It is this sort of result which will encourage continued 
experimentation with a hypertext system until time and experience 
provide a large enough experimental group to have statistically 
definitive results. 

The remainder of Section 4 is divided into four parts. 
Section 4.1 is an evaluation of the computer environment for the 
experiment. Section 4.2 discusses the impact of FRESS on the 
individual user. Section 4.3 looks at the impact of FRESS on the 
course as a whole. The division between 4.2 and 4.3 is somewhat 
artificial since the subject matter of these two sections 
overlaps. Section 4.4 examines the various costs of a system of 
this sort. 4 At the beginning of each subsection we have 
summarized the major findings discussed in that section. 


4. 1 E VALUATI ON OF COMPUTER FACILITIES A ND SOFTWARE 


• There were some serious technical problems and some minor 
ones; while they caused frustration and annoyance among the 
students and instructors, they did not prevent the 
experiment from being successful. Therefore, we can expect 
even better results in a future experiment with these 
difficulties resolved. 


There were minor technical problems with the hardware (the 
IMLAC) and the software (the FRESS program running on both the 
IBM 360 and on the IMLAC) . On the average, difficulties of this 
nature occurred 1.5 times a week to each student. Naturally this 


3 The source for this and later quotes from the instructors are 
informal conversations with project personnel. 

4 The actual expenditures for the experiment are in Appendix D. 
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Computer Facilities and Software 


was somewhat irritating to some of the students, but in general 
did not interrupt their work. 

More serious problems were caused by the Computer Center's 
general purpose time sharing system software on the IBM 360 under 
which FRESS runs. In particular, 360 system "crashes" occurred 
123 times causing a loss of 30 working hours. 31 of the crashes 
occurred during the first week while the Computer Center was 
introducing a new version of the operating system. Although 
system difficulties decreased thereafter, the students' 
complaints did not. Therefore the difficulties in the first week 
appeared to have prejudiced the students' attitudes about 
computers to some extent. Response time — the time it takes for 
the computer to respond to an individual FRESS command — was 
often f rustratingly slow and a major irritant, particularly in 
the afternoons and towards the end of the semester. Student 
complaints about technical difficulties centered on the areas of 
response time and 360 system crashes. Bad response time 
sometimes was both a source of annoyance and a hindrance to the 
students in their attempts to finish specified amounts of work in 
a given time period. Many of their complaints about time 
pressure can be attributed at least partially to this cause. 5 Bad 
response time alsc prevented the instructors from completing all 
of their work in their allotted time slots on the IM LAC. 
Unfortunately, the response time problem was not under the direct 
control of the project personnel. Computer Center staff were 
alerted to the problem but were unable to substantially improve 
it in a single semester's time. 

Admittedly, the technical reliability should be improved. A 
non-e xperimen tal model would have a reduced number of program 
bugs and in all likelihood would not be linked to a general 
purpose computer with problems of its own. The fact that these 
problems occurred on the system we used did detract from the 
positive results achieved. However, it is worthwhile to note the 
following: 1) The problems did not cause people to drop the 
course or prevent them from doing a large quantity of work. The 
benefits were achieved in spite of the problems. 2) The FRESS 
program is designed so that system problems do not cause more 
than the last edit to be lost; other software systems often allow 
users to lose substantial work during such system failures. 

3) "Down Time" before work could be resumed was usually brief. 

4) With the exception of poor response time, system problems 
tended to be clustered for a given student, thus providing many 
trouble-free hours cf work. 


5 System usage 
less severe than 

4 . 2 . 5 . 


statistics show that real time 
the students perceived them to 


constraints were 
be. See Section 
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the impact of FRESS on the 
1) the ease of use, 2) the 
guality of work produced. 


In this section, ae discuss 
student participants in six areas: 
quantity of work produced, 3) the 

4) participant interaction, 5) the time efficiency of the system 
and 6) the building of hypertext associative "networks". 

4. 2. 1 Ease gf U$e 

• The FRESS hypertext system was relatively simple to learn, 
whether or not the user had a technical background. 


FRESS is a 
one or two hours 
-- a user can le 
learn additional 
FRESS. The stud 
an initial one-ho 
supervised for 
room to answer 
weeks. After the 
to teach informat 
students stated t 
after four weeks 
difficulty learni 


layered system. In a relatively s 
of instruction and one or two pract 
arn to do useful work. Thereafter 
features and gradually expand his 
ent participants for this experimen 
ur training session. Then students 
one week. After that an instructo 
questions (which were few) for fiv 
third week there was a second trai 
ion on formatting codes and basic e 
hat they felt fully comfortable wit 
. Only two of the students indica 
ng to use the system. 


hort time -- 
ice sessions 
the user can 
knowledge of 
t were given 
were closely 
r was in the 
e additional 
ning session 
diting . The 
h the system 
ted they had 


As was the case in our first Exxon-funded experiment, 6 we 
found that non- technica 1 people have no greater difficulty in 
learning FRESS than do people with prior computer experience. 
Most of the students in the experiment were liberal arts 
undergraduates, and only two of the students had had substantial 
prior experience with computers. 

In general, we found that students and instructors felt 
comfortable about "talking to a machine" and did not feel it was 
impersonal. This was probably largely because they knew that 
they were actually communicating with people; the computer was 
being used as a transmitting device. 


6 See Section B.4 and A Graphics Jesyonding Tutorial Syst em:_ An 
Experiment for the Exxon Education FQundat ion , Final Report, 
(Providence: Erown University 1974). 
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H-_2 ± 2 _Q uan t i t y_ o f _ H o r k 

• The hypertext group produced prodigious amounts of written 
work, despite the fact that English 16 is not primarily a 
writing course. 


Those members of the Section IB control group who kept their 
notebooks thoroughly (4 out of the 8 students) produced about 50 
handwritten pages (about 25 typed double-spaced pages) , 7 plus the 
final projects, which were approximately 20 pages long, making a 
total of 45 pages. 

Students in a normal section would produce about 4 or 5 
short papers of 4 to 5 pages each and no final project, resulting 
in about 20 typed pages of work. 

We found that the hypertext system encouraged students to 
write more than they otherwise would have. The participants in 
the hypertext section produced an average of 8 4 pages each. The 
students' work varied from a minimum of 57 pages to a maximum of 
137 pages. The average amount of work was about four times as 
much written work as is usually expected in a traditronal course. 
They produced about twice as much as did those students in the 
control section who kept notebooks well. Much of this writing 
was in the form of informal responses that the students composed 
as they entered it into the hypertext. The instructors found 
that these responses, although informal, evidenced comprehension 
and insight* 

This result was one of the most extraordinary, positive, but 
actually unexpected results of the experiment. As discussed in 



Section 4.3 below , ’English 16 is not intended primarily as a 
writing course. Furthermore, the expectations of the 
investigators were in the area of active interaction with the 
material to produce hypertext trails (Section 4.2.6). In fact, 
because the work was done online, and there was no printed output 
during the course, we were unaware of the amount of writing being 
done until the bulk of the course was completed. Then when we 
became aware of the expanding requirements for storage, we 
computed the actual amount of material produced. During the 
course, our major encouragement was from the excellent quality of 
the work and from the benefits derived from having the students 
see each other's work. (See Sections 4.2.3 and 4.2.4.) 


7 For an explanation of the use of notebooks in the control 
section see Section 3.3. The other 4 students completed their 
notebooks sufficiently to pass, but did substantially less work. 
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The difficulties in getting students to write well are 
widely publicized. Writing is the sort of skill which requires, 
among other things, practice in simply producing large quantities 
of work. The enormous output of the FRESS section students 
suggests future use of the system as an aid in the instruction of 
wr it ing. 

4 i 2 i 3_Quali ty_of _Work 

There are three sources of information concerning the 
quality of the work of the students in the hypertext section: 
1) grades for the course, 2) the comments of the instructors (and 
to a lesser extent of the students) , 3) the independent 

pre-course and post-course tests evaluated by an independent 
examiner. 

4.2.3. 1 Grades 

• Because of the small sample and the tendency of instructors 
to grade "on a curve," the analysis of grades produced 
inconclusive results. 



Probably the least information is given by the grades. 29% 
of the students in English 16 and 33% of the students in the 
hypertext section did not take the course for a grade, but 
elected Brown's "satisfactory/no credit" option. This masks the 
grade distribution. The grade distribution of the hypertext 
section was approximately the same as for English 16 overall. 
This is the result that one would expect since instructors report 
an unavoidable tendency to grade "on a curve" -- using 
comparisons between students in a given section as a basis for 
selection of grades. To the extent that this is true, one could 
expect to gain little information by comparing grades between 
sections. Grades are included in Appendix C. 1. 

4. 2. 3. 2 Instructor Evaluation 

• The instructors all felt that the students made significant 
progress that can be attributed to the use of the hypertext 
system. Major improvements were in the areas of increased 
self-confidence and improved critical abilities. 


The hypertext section students were closer to being 

"finished", accomplished readers of poetry than the other 
students. First, they could integrate secondary material better. 
This was, in part, because they used more of it -- but the 
hypertext system was what gave them the opportunity to accomplish 
so much. 
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In a normal section, the students tend to memorize "correct" 
responses to poetry, rather than learning to become critics 
themselves. But the hypertext system put a premium on the 
ability of the student to react to the poem alone, to the 
secondary material alo ne, with instructor direction occurring 
later. Also, it produced a strong attitude or opinion in the 
student before he or she encountered professional criticism, 
which helped in avoiding intimidation by the critics. 

Second, the students became far more confident of their 
critical abilities: 

One of the things that we accomplished that is so 
rarely accomplished in poetry courses that it's almost 
an embarrassment, was that we, in most cases, freed the 
students of fear of poetry. Their gains in confidence, 
which show in the way they wrote about poetry, in the 
way they talked about poetry, were extraordinary.... 

For me, one of the most revealing incidents of the 
whole course was when I read a student's response in 
which the students were asked to look at the responses 
of professional critics to a poem and also look at 
their fellow students' responses. One of them said, 

"You know, several of the students in the class said 
things about this poem that were just as helpful to me 
as anything I read in the professional critics." Now 
when we reached that point I thought we had succeeded, 
not just because one student had made a very good 
response, but because the other student had realized 
that a student cou^d make a response that was just as 
valuable as the responses made by professional critics 
to the poem. For me this was an extraordinary moment 
and I think a very important one and it says something 
about what can be accomplished with this kind of course 
that can hardly be accomplished in any other way. " 8 


A large part of the writing done was not polished prose. 
This was because the object of English 16 is to expose the 
student to a large quantity of different sorts of poetry. The 
emphasis was on reading, net writing, so polished prose was not 
expected. The system is capable of being used to teach 
composition if that were the goal. 



8 Professor Robert Scholes during a discussion. 


- 17 - 


R esults 


Individual Usage Patterns 


4. 2. 3. 3 Independent Evaluation 

• Although we can not claim definitive results because of the 
small sample involved, the independent evaluation showed 
students in the hypertext section to have improved more than 
students in the other sections. 


The independent evaluation was the most objective 
measurement of quality available. The students from each section 
of English 16 responded to one poem before they took the course 
and to another after completion. Professor David H. Hirsch of 
Brown's English Department, a former teacher of English 16, 
evaluated these two responses for each of 44 students completing 
the course. Professor Hirsch and the course instructors selected 
poems that they felt were equally difficult to analyze. Each 
student response was typed and encoded with a number. This was 
done to ensure that the responses could not be identified by 
author and "before and after" responses of an individual could 
not be connected by Professor Hirsch. He graded each comment on 
a scale of 1 to 10. He looked for 1) correctness of 
interpretation, 2) ability to understand metaphorical language 
and its place in the poems, 3) ability to recognize the 
techniques of writing poetry - imagery, mood, tone, 4) ability to 
perceive the nuances and subtleties of the poem. 

The results of these evaluations are summarized in the 
following table. The data on which the table was based is in 

Appendix C.2. The hypertext section is 1A, the control section 
IB; 2 and 3 are the other English 16 sections. 



Table 

Section 

2: Evaluation Results 
1A IB 

2 

3 

Average before 

6.83 

5.8 6 

. 73 

7.08 

Average after 

8.00 

6.8 7 

. 26 

6.58 

Improveme nt 

1. 17 

1.0 

. 53 

- .50 

K umber improving 

8 

4 

8 

3 

By 3 points 

3 

0 

1 

1 

By 2 points 

3 

1 

3 

1 

By 1 point 

2 

3 

4 

1 

lumber declining 

1 

0 

3 

6 

Dumber the same 

3 

1 

4 

3 

Number evaluated 9 

12 

5 

15 

12 

Number enrolled 

12 

8 

22 

16 
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It should be noted that the scores attained on these tests were 
high: no score below a 4 was given. Although he was quite 

prepared to assign any value from 1 to 10, Professor Hirsch 
found the students to be unusually articulate. 

These results have one peculiar feature: in ten cases, 

students showed a decline. That is, the work done after the 
completion of the course was judged to be poorer than that judged 
when the course began. There were several "spectacular" 
declines. For example, the only student whose analysis merited a 
10 before the course received a 6 on the post course analysis. 
One of the individuals who declined was in the hypertext section. 
That individual went from a 9 to a 6. That same student received 
an A in the course and was judged by the instructors to be one of 
the best students in the experimental section. The number of 
English 16 students who declined thus represents 22.1% of the 
students who took the test. We are not able to adequately 
explain this puzzling phenomenon. 10 The numbers involved do not 
allow any significant statistical analysis of this decline. 

Having said this, we nevertheless feel the results are quite 
encouraging. Of the four sections evaluated, the hypertext 
section ranked second in pre-course analysis, and first in 
post-course analysis. Hypertext section students averaged the 
greatest improvement. Their average improvement was 1.176 
points. The control section had an average improvement of 1 

point. Section 2 improved .53 points. Section 3 showed an 
average decline cf .5 points. Section 3 had six of the ten 
persons that declined. 

Although they comprise 27% of the students evaluated, the 
hypertext section contained 60S of those who improved by 3 or 
more points, 37.55? of those who improved by 2 or more points and 


10 One possible explanation, suggested by Prof. Hirsch, was that 
although there was an attempt to have the poems equivalent, in 
fact, they were not. Thus, the second poem may simply have been 
more difficult to judge than the first. Another explanation has 
to do with the "fit" between particular poems and particular 
readers. It is impossible to find poems that will be of equal 
ease or difficulty for a group of readers. A third possibility 
that cannot be overlooked is that some traditional courses in 
poetry have a negative effect on the student's longtime ability 
to respond intelligently to new poems. In fact. Prof. Scholes's 
suspicion that this third possibility is true was one of his 
motivations for participating in this experiment. The only way 
to find the correct explanation is to have more evaluations of 
this sort. See the comments at the beginning of Section 4. 
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34% of those who improved at all, 27% of those who stayed the 
same, and only 10% of those who declined. 

Although more data needs to be collected, and more 
information is needed to determine why the "declines" occurred, 
the results are encouraging. 

4 ^2^4 _i n terac tion 

This section will discuss two areas: How the students 

interacted with the stude nt- wri tten material and how the 
participants interacted with each other. The FRESS hypertext 
system allows both types of interactions to occur in various 
ways. Information on interactive usage patterns came from 1) 
command usage statistics, 2) inspection of student entered 
material, 3) user reports. It was a goal of the experiment to 
encouraqe use of both forms of interaction and test their 
desirability in education. This goal proved to be overly 
ambi tious. 

4. 2. 4. 1 Interaction with the Material 

• While active interaction (see below) with the material was 
not in the form that we expected, it was extensive and a 
great improvement over our earlier experiment. 



There are two sorts of interaction with the material, which 
we may call active and passive. Passive interaction is reading 
the material and, where jumps 11 (links) are in it, following 
these jumps to other related material. Active interaction 
involves changing the material by adding links between existing 
items of text, adding additional material, and adding links 
between the user-authored material and pre-existing material. 

Passive interaction with the hypertext was generally quite 
extensive. Students tended to follow pre-established hypertext 
links in the material, both those established by the instructors 
in the initial data base and those added by participants during 
the course- We learned that it is necessary to provide more 

information concerning what is at the other end of links 
(especially guantity) at the point where the link would be 
followed; students tended not to follow links if they did not 
know what they would be reguired to read. This use of passive 
hypertext links confirmed what we found in the Exxon experiment. 

The active use of hypertext was minimal during the Exxon 
experiment, and we were determined to improve that aspect during 


i > See Appendix B for an explanation of this term. 
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the current experiment. To accomplish this we provided the 
students with a strong incentive to become active users of the 
hypertext at the beginning of the course: we required the 
students to enter responses to poems and other materials at each 
session, starting with the very first session each student had. 
This helped them to overcome any initial hesitation to alter the 
instructor-provided text. We hoped that they would subsequently 
make use of additional features on their own initiative. The 
major effect was, instead, the enormous output of work discussed 
in Section 4.2.2. Since these responses were required, this form 
of active interaction with the material can be called 

non- voluntary . Another result was that students became more 
aware of what they were learning: 

I think one of the beauties of this system is that one 
is constantly making discoveries and is aware of the 
learning process. Having a full explication available 
would change the sensation of accomplishment to one of 
mere comprehension. 12 

Voluntari active interaction with the material, however, was 
still not as extensive as we would have liked it to be. 

It now seems that, if voluntary active interaction is 
desired, incentives probably should be restricted to the early 

part of a course. In this course the assigned responses were 
continued throughout the semester. Consequently, students did 
not always have sufficient time to browse, to ruminate, and to 
ask questions and make comments as much as they desired. If 
students are to use the hypertext to full advantage they should 
not be given assignments that fill up their time completely. 13 

Further discussion of this problem is in Section 4.2.6. 

4. 2. 4.2 Participants' Interaction With Each Other and With 
Instructors 

* Interaction of various kinds among the participants was 
perhaps the most useful aspect of the hypertext system. 



Our initial emphasis in this experiment was on enhancing 
interaction between the students and the reading material for the 


12 Student "online" reply to suggestion that the instructors 
provide definitive interpretations to each poem. 

13 Poor response time may have contributed to this problem. The 
assignments might not have been quite so overwhelming had 
response time been good. See Section 4.1. 
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course. As discussed in the previous section, FRESS encouraged 
this form of interaction in a successful and stimulating manner. 
We found that it also produced unexpectedly valuable interaction 
among the participants through the medium of hypertext. 

There were several varieties of such interaction possible 
for participants in the course. One was made available by the 
insert comment ( . i c ) and insert question (-ig) command macros 
which could be applied at any point in the material being read. A 
"tag" would then be inserted at that location in the text to 
point other users to the question or comment. Users could then 
choose whether or not to view this marginal note and perhaps 
respond to it in the hypertext. Readers could also go directly 
to a file containing the comments and take links "backwards" to 
find the context of a comment. 

A second type of interaction, that between student and 
instructor, was available through the assigned responses. 
Students did their work using the system, and instructors read 
student responses and made evaluations, again online. 

A third type of interaction, between student and student, 
was also available through these assigned responses. Students 
often read the responses of their colleagues, a sort of "passive" 
interaction. Several times students were assigned work which 
required them to read and ev^iuaie the responses of the other 
students and instructors. This was an "active" form of 
student-student interaction. 

A fourth type of interaction was available among the 
instructors. They were able to see and evaluate each other's 
work online in a way not normally possible. 

Of these types of interaction, the first, using comments and 
questions, turned out to be the least important. 15 This may have 
been partly due to the fact that too much work was assigned. The 
other three types of interaction turned out to be very important. 

Interaction of type 2, involving instructor comments on 
student responses, was, of course, of critical importance 
throughout the experiment. It was the primary means of 
communication between students and instructors, replacing graded 
homework assignments and, to some extent, classroom discussions. 
It provided the instructors with more material than usual on 
which to base their evaluations of the students. At the same 


1 *See Appendix B. 2 for a description of how hypertext is used. 
15 This interaction is related to the active use of hypertext 
discussed in Sections 4. 2. 4.1 and 4.2.6. 
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time it provided the students with more feedback than they can 
usually get in a traditional course. 16 

Students made extensive use of the passive variety of 
student-student interaction (type 3). They almost always read 
the comments, questions, and responses left by other students and 
found them both enjoyable and helpful. The students felt that, 
even though they had less classroom contact, they grew to know 
each other well intellectually because they had so much contact 
through the hypertext. The comments of the better students 
provided instruction to ether students without any additional 
effort on the part of the better students (or the teaching 
staff). Being able to view the thought processes of other 
students in coming to grips with the material was nearly always 
instructive, regardless of the guality of the other student's 
conclusions. Moreover, the value of the system was not 

restricted to the better or the poorer students; all levels of 
students benefitted from interacting with the growing manuscript 
material. 

As with student-material interaction, there was a 
disappointingly lew amount of "active" student-student 

interaction: direct reference to or critiguing of the work of 

other students. When "active" interaction of this kind was 
required (e.g., "Choose one other student response and compare it 
to your own") the results were excellent. However, there was 
little interaction of this kind when not required. Again, we may 
attribute this substantially to the time pressure to finish 
assigned responses. 17 

The instructor-instructor interaction (type 4) was largely 
unexpected. It resulted in an improvement in the quality of 
instructor comments: the instructors stated that since their 

comments were to be viewed by all the students and the other 
instructors, their responses tended to be better, more complete 
and carefully stated. The instructors sometimes disagreed over 
particular interpretations and indicated their disagreement in 
the hypertext. This provided the students with a different 
perspective on instructors than is usually available, since 
courses of this nature typically have only one instructor and 
therefore no chance for such discussion. As an additional 


16 Despite this greater amount of feedback, a few of the class 
members felt that they did not know "where they stood" or how 
well they were doing as the course progressed. This may have 
been because of the lack of any strict judgmental guality in 
instructor comments. That is, instructors did not grade the 
responses, but instead critiqued their contents. 

17 See Section 4.2.4. 1 for a discussion of the same phenomenon in 
student-material interaction. 
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benefit of this type of interaction, the primary instructor. 
Prof. Scholes, knows much more about the abilities of his 
graduate students to teach than is normally possible. 

4. 2. 5 Use o f lime 

• The hypertext system encouraged efficient use of time and 
did not require inordinate amounts of time to do the 
assigned work. 


The benefits of the hypertext did not require unusual 
amounts of time from the students. During the months of October, 
November, and December computer time was spent primarily doing 
work on the required sessions, although the term project had been 
assigned. Brown University's Christmas vacation extended from 
December 18 until January 5. Class assignments were discontinued 
from then until the end of the semester on January 23. During 
the three week period from January 5 to January 23 there were no 
classes and the students worked exclusively on their final 
projects except for those few students who had fallen slightly 
behind schedule. 

During October through December, while doing assigned 
sessions, the students averaged 5.7 hours per week. The time 
spent by each student was fairly steady throughout these months, 
since weekly assignments discouraged delaying work. For the 
entire three weeks in January, the students averaged 14.73 hours. 
The weekly average would be 4.9 hours, but this is misleading 
because students were free to schedule the necessary hours at any 
time during this period. 

The students reported doing little work on the course 
outside of time spent at the computer terminal. Interestingly, 
they reported working, on the average, 7.2 hours a week, 
significantly more than the 5.7 hours actually spent. Although 
5.7 (or even 7.2) hours is less than is usually expected in 
English or most other college courses, two-thirds of the students 
also reported spending more time on this course than on any other 
course. Perhaps that was true, but it seems possible that just 
as the students overestimated by 1.5 hours the amount of time 
spent on the computer, they also mistook the concentrated effort 
for this course as working a greater amount of time than in other 
courses. That is, the students' wcrk output (Sections 4.2.2 and 
4.2.3) supports the view that they worked harder than in a normal 
course, but this work did not take more time than in a normal 


18 Prof. Scholes points out that the data may also mean that 
students work less than is assumed in a normal course. In other 
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course. 18 He may therefore conclude that FBESS time is time 
efficie ntly spent. 

In addition to their overestimation of time spent, the 
students underestimated the amount of free time that was actually 
available for their use. Brown's computer was available 114.5 
hours a week during the fall semester (Monday, Wednesday, 
Thursday, Friday, 7:30 A.M. - 2 A.M.; Sunday, 9 A.M. - 2 A.M.; 

Tuesday, 10:30 A.M. - 2 A.M.; Saturday, 8:45 A.M. - 4:45 P. M. ) . 
During October, November, and December, utilization of the system 
was 73% and during January 54%. The students reported 

occasionally feeling rushed, but the system utilization figures 
do not support the view that system availability was the cause of 
this pressure. (The percentages just cited include instructor 

utilization and take into account the time the system was not 
available because of IBM 360 "crashes".) If one discounts time 
rarely used, after 7 P. M. on Friday night and before 2 P.M. on 
Sunday morning, then Fall utilization was 79.5% and January 
utilization was 60%; both figures show significant time to spare. 
He should note that, of course, students' lives are not 
completely flexible. We feel, therefore, that twelve may be the 
maximun number of students for a course such as this with the 
facilities currently available. 

4.2.6 Build ing H ypertext Networks 

• Students have not been completely successful in using 
tr ail- bu ilding effectively. The reasons for this are 
complex and not yet fully determined; we have, however, 
learned valuable lessons which can be applied in future 
experiments. 


Vannevar Bush's notion of having users "blaze" new trails 
through the hypertext is an exciting concept. 19 We have 

attempted to encourage this type of use in the NEH and Exxon 
experiments. The final project of this experiment (see Section 
3.2) was designed expressly for this purpose. Nevertheless, 
students have not been completely successful in using 
trail-building effectively. There seem to be two factors 
involved. One is discussed here; the other, the inappropriate 
nature of the final project assignment, is discussed in Section 
4.3.4. It will take future experimentation to isolate the 
determining factor. 


words, the students' proportions are accurate and our notion of 
"normal" time spent on courses is incorrect. 

19 See Appendix B.3. 
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We have learned from these experiments that trail-building 
is a complex task that requires significant prior planning to 
integrate its use properly into the course structure. Use of the 
trail-building facility must be a natural outgrowth of the needs 
of the course and cannot be imposed arbitrarily. Also, it will 
not be applicable in all cases. 

We must distinguish the ability to make trails from the 
skills mentioned in Section 4.2.1 above. We are not talking of 
the skills reguired to operate the terminals and perform the 
various browsing and editing functions available in FRESS. We 
are referring here to the skill to effectively employ the FRESS 
features to structure a body of computer-based manuscripts. The 
problem is one of "perceptual skill". The students must adapt to 
a new medium; they must learn to think in a new, nonlinear 
fashion. They must learn to find their way in a large mass of 
information that they are building up and structuring without the 
familiar physical cues provided by books (position on page, 
position of the page in the volume, color of binding, position of 
book on shelf, etc.). 

We know that this adaptation is possible; the instructors 
who helped construct the hypertexts for our two experiments and 
people who have used and built hypertexts for several years have 
mastered this problem. Apy difficulties that may be encountered 
in acquiring the skill is, for many, offset by a hypertext's 
advantages; ready access to a large body of material, ability to 
create non-arbitrar y connections, ability to rapidly and easily 
change and expand any previously existing structure. However, it 
seems to take longer than a single semester to become comfortable 
with this new way of seeing information. We have yet to 
delineate the skills required for this mastery or the methods 
needed to teach it. He suspect that, contrary to our original 
expectations, an initially small set of materials is probably 
more conducive than a large set to the instruction of 
trail-building in the context of a single course. 

Another possibility is that before one can master this 
skill, one needs experience at building a personal hypertext. We 
hypothesize that a user should first practice building hypertexts 
using familiar subject matter and research tools such as books 
and journal articles. Once the skill is mastered, it should be 
transferable to less "familiar computer-based hypertext. This 
exercise might provide the user with a feeling of control over 
the material that would help in maneuvering through a hypertext 
in the complex way required for trail-building. 
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4.3 COURSE DESIGN CONSIDERATIONS 


Before t be course started, the instructors had a clear idea 
of some of the benefits they thought would result from the use of 
a hypertext system. They planned the course accordingly. Many 
of the expected benefits were realized. Primary among these was 
the value of having each student in the course give a response to 
the poetry at various stages of exposure to related materials. 
In particular. Prof. Scholes felt it was important for the 
development of a student's independent critical abilities to make 
a response to a poem before ajjy exposure to other critics 
(including other students). The system was utilized to force such 
a response while being selective about the amount and kinds of 
other materials available to the student. As a result, the 
students' critical skills and appreciation of poetry improved 
dramatically (see Section 4.2.3). 

One benefit that should not be overlooked is the ability of 
the instructors tc monitor closely the activity of all involved. 
As Prof. Scholes indicated: 

Given the kind of system we have we know who used what 
and how much they used it and what type of use they 
made of it. We also know what kinds of comments all 
the instructors involved made in responding to the 
students. We just knew much more about what happened 
than we ever know what happens in any ordinary course. 

And I have a feeling that if we knew what happens in 
some courses that weren't taught on this system that we 
would be appalled at the actual results of those 
co urses. 


There were four results of the experiment which the 
instructors determined would change the way they utilized 
hypertext in the future. These were in the areas of 1) student 
interaction, 2) classes, 3) quantity of work assigned, and 4) the 
nature of the final project. 

4. 3. 1 Interac ti on 

• Interaction among the participants in the course was more 
important than originally envisioned; this form of 
interaction would be more strongly emphasized in future 
e xperiments. 



The interaction among the participants (see Section 4. 2. 4.2) 
is significant when considering future uses of this system. The 
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importance of this form of "active" interaction was unexpected, 
as Prof. Scholes makes clear: 
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When students are asked to comment on each other's work, one 
quickly finds out what they know by what they do or don't notice. 
The hypertext system put their written work on the same level, to 
some extent, with the materials written by professionals. This 
encouraged the students both to think of their own work as more 
valuable and to think of the work of the professionals as more 
susceptible to criticism. Manipulating hard-copy pages to do 
this sort of thing manually is guite difficult for the 
instructor, if not impossible. 


4. 3. 2 Classes 


The benefits from classroom and hypertext experiences are 
complementary. Both are needed in the course format and 
neither should be emphasized tco strongly at the expense of 

4- L ~ ~ 4- U ^ — 


the other. 
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education thus complement each other. 

4^ 3_2iid.Qtit y of Required Course Hoik 

• In this experiment the students spent most of their time 
doing required reading and responses. If instructors of 
future courses wish to include more writing of polished 
prose and more interaction among the participants, then less 
predetermined work should be assigned. 


While the students demonstrated that they were able to do a 
large amount of work in the time provided, the instructors felt 
that perhaps too much material was provided, occasionally 
assignments were too long and toe complicated, and some of the 
material was unnecessary. Ihere are obvious benefits to having a 
great deal of material available. Host of these are oriented 
around those goals of the course that emphasize student-material 
interaction. The other benefits of the system, student- student 
interaction and the encouragement of writing, seemed so great 
that xn the future the instructors reported that they might limit 
the quantity of material so as to concentrate more on achieving 
these other goals. 


h. 3. U Final Pr oje ct 


• The final projects did not successfully encourage effective 
trail-building. This may be partly due to the mismatch 
between the nethcdclogy taught in the course and the 
techniques reguired to write the final papers for the given 
assignme nt. 
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We did not decide to do a final project until after we had 
organized the rest of the course. It was an attempt to encourage 
trail-building (see Section 4.2.6) that would not otherwise have 
been done given the structure of this course. In retrospect this 
was a mistake. The projects ranged from good to bad, but were 
mostly poor. The project reguired the student to view the poetry 
in terms of the historical development of a particular poetic 
theme or image. The course itself had not considered poetry in 
such terms, although a chronological progression had been 
followed. Problems occurred primarily because the aspect of 
historical progression was not emphasized as strongly as that of 
stylistic or thematic similarities among poems. The comparisons 
drawn in each unit among large numbers of poems from all time 
periods apparently disrupted the underlying chronology of the 
major poems of each unit. As a result, students experienced 
difficulties in developing a progressive theme and instead 
resorted to exemplifying their thoughts in a somewhat random 
fashion. A better final project would have required the critical 
response skills the course had emphasized. The anthology used by 
the control group as a source for their papers was organized more 
obviously chronologically and the approach used in their classes 
was oriented in a similar manner. This might help explain why 
their papers were mere clearly structured. 

The hypertext system seems, overall, to have neither helped 
nor hindered the writing of the final project. Having material 
available at their fingertips was helpful in choosing sources for 
the papers. On the other hand, the scheduling restrictions and 
the problem of never seeing more than a "window" of text in the 
paper at any one time presented substantial difficulties. These 
problems did not adversely affect the writing process for the 
informal responses. Writing a polished paper, however, does seem 
to require some combination of hard-copy printouts with online 
composition and editing. This capability already exists in FRESS 
and will be utilized in the future. 

4.3.5 E x porta bi li t y 

• while there is possibility of transfering some aspects of 
the system to other institutions, the hypertext developed 
for English 16 would not be useful ir. a more traditional 
f or m. 


One question of importance when dealing ■ ith a prototypical 
experiment of this nature is whether the courseware developed for 
the experiment can be transported and used elsewhere. If FRESS 
or a similar hypertext system were available somewhere else, the 
English 16 hypertext could certainly be used by other students 
and instructors. The critical question, however, is whether the 
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hypertext materials could be used in some other, more traditional 
form. He feel that the answer, for the current experiment, is 
no. The hypertext consisted almost exclusively of pieces of 
primary and secondary source material with little commentary or 
direction from the instructors beyond prompting questions at each 
session. The impressive results of the experiment arose from tb.p 
subsequent interaction among the participants, a feature that 
could not be satisfactorily duplicated with workbooks or some 
other traditional approach. This is not to say that no hypertext 
could be transformed into similar, suitable but non -hypertext 
teaching materials. In fact, future experiments that have been 
tentatively planned are intended to provide for some form of 
exportability, either of the results or of the hypertext itself, 
as teaching material. For example, higher level seminar courses 
could be oriented towards the production of polished manuscripts 
for publication rather than just informal responses. 


4. 4 COSTS 


4._ 4_. J_COK£U t££_CCSt S 

Computer costs (using Brown’s internal computer time 
accounting scheme) which were directly related to the course may 
be summarized as follows: 



cost 

per 

hour 

Student use (934 hours) 

$3804 

$4.07 

Instructor use (145 hours 20 ) 

785 

5. 41 

Storage disk rental 

2385 


Total (1079 hours) 

6 974 

6.46 


This hourly cost of $6.46 compares favorably to an average of $10 
for scientific student interactive use. In the Exxon Education 
Foundation experiment (Appendix E.4) the hourly cost was $8. For 
12 students, the Exxon experiment had a student use of only 571 
hours (61% of the current experiment's student use), but the disk 
storage costs are largely fixed. Therefore, the principal reason 
for reduced hourly costs in this experiment was the use of the 
disk over a greater number of hours. The non-disk per hour costs 
were approximately the same (Exxon, $4.14; NEH, $4.07). 


20 This is time spent by instructors during the course and does 
not include instructor use during course preparation or other 
staff use at any time. 
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4.4.2 Precourse Preparation 

As might be expected, the instructors spent more time than 
usual setting up the material for this course. They estimated 
this time to be twice what is rormally reguired of a new course 
in poetry and four to five times what they would expect for a 
normal course in poetry. Since the instructors were already very 
fa mi liar with the material, somewhat less time than usual was 
reguired tc prepare for the once-a-week discussion class. This 
preparation time and some of the time normally spent in the 
classroom were instead devoted to interacting with the students 
through the hypertext. All three instructors together averaged 
12.7 hours a week on the computer system. As instructors become 
more familiar with the system it seems likely that less time will 
be reguired; however, a course of this nature will probably 
always represent a greater commitment from the faculty than is 
demanded by a traditional course. It is unrealistic to expect 
students to produce more in a course like English 16 without 
greater instructor involvement. Again, Prof. Scholes makes this 
point: 

I doubt seriously that it's possible to get more work 
from students without faculty putting in more work to 
meet that. Because the students have written so much, 
we have had to read so much and write so much in 
response to the students. If our comments to their 
responses had fallen short at any point, had been 
inadeguate or seemed perfunctory, I doubt if the 
caliber of their responses would have continued to 
improve as it has over the course of the semester. The 
course, in that respect, demands a great deal of time. 

What we save on classroom time we put back in response 
time on the system. ...My suspicion is that we cannot 
expect to save much instructor time with this kind of 
course. The gains are not to be found in that area. 

We can expect to teach much better and to produce 
students who are better readers of poetry and feel more 
confident about poetry. I think that’s a sufficient 
justification for what we've done. 
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5 CONCLUSION 


He have now determined conclusively that a hypertext system 
provides substantial educational benefit to students and 
instructors. A hypertext system provides a convenient medium for 
encouraging the occurrence of, and magnifying the impact of, 
student to instructor, student to student, and instructor to 
instructor communication. Students using a hypertext system have 
an excellent per hour productivity. They do more work and learn 
more than students using traditional educational methods. The 
result is a significant new approach to the development of 
critical and writing skills. 

Although we now have a positive, experimental basis for 
encouraging wider use of hypertext in education, much remains to 
be done in understanding how to apply this technique for maximal 
benefit. The results of this experiment have provided us with a 
list of issues to be explored. These include such items as 
investigating the impact of such systems on students' and 

instructors' time, and delineating guidelines for using such 

systems for a variety of courses and educational settings. 

Moreover, certain puzzling aspects of the experiment should be 
followed-up. For example, while student use of the hypertext was 
much more extensive than in earlier experiments, the students 
still did not develop their own hypertext trails and text 

interconnections as much as we had hoped. 

Two projects have already been proposed to assist in 

exploring some of these remaining issues. The first project 
would attempt to apply two of the most positive results of the 
current experiment -- the quantity of work produced and the value 
of interaction -- to graduate level seminars in English and 

Philosophy. The second project is more limited and would be a 

repeat of the current experiment, but with certain modifications. 

This would enable us to validate the instructional benefits 
perceived in the current experiment with additional data. In 
addition, a better measure of instructors' time could be 
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To disseminate the results of the experiment we have 
prepared a twelve page description of hypertext and its 
educational uses, including a discussion of the format and 
results of the current experiment entitled, "Computers and 
Poetry." This was distributed to people both inside and outside 
the University who expressed an interest in the project. We have 
had several additional reguests for copies of this document since 
then. When it was published, a copy was sent to NEH. A second 
copy is enclosed with this final report. 

Dr. Karl Zinn made a presentation entitled "Computer-Based 
Poetry Seminar and Its Evaluation" at the Conference of the 
American Educational and Research Association, San Francisco, 
April 23, 1976. Copies of "Computers and Poetry" were made 

available at this conference. 

We will be sending a notice to the people on our project 
mailing list indicating the availability of this final report. 

The major effort in publicity will be based on the film 
about the experiment that was produced as part of the project. 
The film was made by Michael Silverman of Brown's English 
Department, Peter O'Neill, and Bill Gallery. It includes remarks 
by Professors van Dam and Scholes and discussions with several 
students from the English class. Final prints of the movie are 
now being made; a copy will be sent to NEH as soon as the prints 
are finished. 

We intend to show the film at the Modern Language 

Association National Convention in December, 1976. Followup 
presentations will be made ^y several project members who will 
attend the conference. The film will also be presented at 
conferences concerning computer-based instruction. 

Papers on the technical aspects of FRESS used in the course 
are also in preparation for publication in CAI and computer 
science journals. 
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Bil_lHE_CCNCEPT_OF_HYP£RT EXT 


A hypertext 21 is a new manuscript form that consists of a 
computer-based collection of related documents together with all 
their cross-references, footnotes, annotations and other 
associations. The Encyclopedia Britannica, the annotated King 
James Bible, and Nabokov's Pale Jire are examples of manuscripts 
containing information organized in a form similar to the 
organization available with hypertexts. Another example, drawn 
from the material available to the students in the experiment 
reported here, might be pictured as follows: 


(A) , T 

I JCH N DCNNE j 

| %J 1 The R el ique | 

I %J2 Biographical l 

| information ... I 

I % J3 Critical | 


| Art ic les. . . | 

L 1 


(B) , 

!%P1 The Religue | 
| When my gra ve is| 
I broke up againel 

, , 

I 1 

, , 

i j 


(D) r 1 

|%P 2 Biogra- | 

I ph y of I 

! Donne j 


I 3 A discussion of 
I Imagery in. . . 


, 

i 

1 ^J3. 1 Additional 
I Discuss ion 

I 

L ! 

(E) r 1 

|XP3. 1 from discussionl 

I of imagery. . . I 

I , 


I 


This diagram represents part of a hypertext: it consists of a 
table of contents showing some of the available manuscripts (A), 


21 In addition tc Theodor Nelson's, "Getting It Out of Our 
aysten," o_p- cit. , see his C cm cute r Iib/J)ream Machines, Hugo's 
Book Service, Box 2622, Chicago, Illinois and "No More Teachers' 
Dirty Looks," Computer decisions , Vol. 2, No. 9 (1970) , 16-23. 
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a poen (B), other primary source material (e.g., a biography of 
the poet) ( D) , and secondary source material (e. g. , critical 
art icles ) (C, E) . 2 2 

Pursuing cross-references through a large collection of 
manuscripts in a library in the traditional manner is not always 
a simple task. In a hypertext viewed on a computer-connected 
television-like display console, however, a reader may follow any 
cross-reference simply by using a pen-like device to point at a 
link from one text fragment to another. The computer then brings 
the indicated text fragment instantaneously into view. 
Furthermore, the computer remembers the succession of links that 
were followed to allow backtracking. Related items may be 
chained together easily and left as trails for future browsers. 

These facilities provide unprecedented maneuverability 
through a complex web of information and associations. The path 
taken through the hypertext at any given time is determined 
entirely by the reader. Thus, unlike other computer-based 
informational systems, a hypertext can serve as both a retrieval 
mechanism and as a browsing environment. More importantly, the 
organization imposed on the information by the hypertext is not 
static; each reader can add new associations and/or reconfigure 
the old ones into new patterns. 

At Brown we have developed a computer-based tool called 
PRESS (File Retrieval and Editing System) for use in educational 
and research settings. In particular, FRESS combines text 
editing capabilities, used in the past primarily to aid in 
manuscript production, with hypertext facilities. These hypertext 
features provide for cross-referencing material, entering 
questions and comments, and for ke y word -ba sod retrieval of 
information, all under user control. They allow a user -- either 
student or instructor -- to organize material in a non-linear and 
non-prescribed way and to change that organization at any time. 
It is not merely the organization of material that gives FRESS 
its special value, since some organization is imposed on all text 
materials. Rather, FRESS is a powerful new tool because it 
provides the ability to alter the organization as well as the 
material so organized. 


22 The i J's and %F's are special codes indicating the beginnings 
and endings of "links" between text fragments. 
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Tne computer program designed at Brown to implement the idea 
of hypertext allows a student, teacher, or researcher to sit 
before a display console screen similar to a television screen. 
In one of three different "windows" outlined on the screen a 
portion of text may appear before the user. 23 To advance the 
text tae user taps a button on a keyboard causing the text to be 
rapidly replaced by the next "page" or window of material. Let 
us suppose that a student were browsing through the material 
outlined schematically above. If the reader were looking in the 
directory of material related to John Donne it might appear as 
fo Hows : 


r ~ i 

I JOHN DCNNE 
| %J 1 The Religu e 
I %J2 Biographical information 
I % J 3 Critical Articles 
I %J4 Other poems by Donne 
I 

i r 

I i 

J I 

I I 

I I 

I I 

I I 

! L 


Suppose the reader wished to look at "The Religue". By pointing 
a light sensitive pen at the link or "jump" to the item 
(%J1 The Religue), the poem would appear instantly in another 
wi ndow: 


23 The screen may be configured in seven different ways to get 
from one to four windows of different sizes. Proportions shown 
here are not accurate; the screen is 21" square and is capable of 
displaying 3000 characters. 
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, | 

| JOHN DONNE | 

| % J1 The Religue | 

I %J 2 Biographical information I 

I %J3 Critical Articles I 

| % J 4 Other poems by Donne I 

I | 

i , j 

| John Dcnne: The Relique | I 

|When my grave is broke up j | 

I againe | I 

| Some second ghest to I I 

| entertaine, XJ5 | I 

I I I 

I 1 ! 


The second line of the poem contains a marker ('J5) that 
indicates that the instructor -- or another student -- has 
included some explanatory material about that line. The reader 
may wish to follow this jump to the explanatory material. By 
pointing at the jump (%J5) with the light pen the annotation 
would appear instantly in another window: 


1 "■ “ ' - 1 “ 1 ■ 

I JOHN DONNE 



1 

I %J 1 The Religue 




I %J2 Biographical information 




1 %J 3 Critical Articles 




| % J4 Other poems by Donne 

i 

J _ _ _ 



J 

9 

IJohn Donne: The Religue 

i 

I % P 5 Second guest: It 

was 

1 

1 When my grave is broke up 

|a common practice in 

the 


| againe 

|17th century to re-use 


1 Some second ghest to 

1 a grave after an interval 


1 enter taine, %J5 

1 

i _ _ 

1 of several years. 

1 

l 


J 


In a similar fashion the reader could follow the reference to the 
biography or critical articles or continue reading "The Relique" 
or one of the other poems. The reader would proceed, following 
an associative trail of items at his own leisure, lingering on 
some items and browsing through the larger works containing them, 
investigating commentaries and controversies as they are 
mentioned or skipping over them to more salient items. The 
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reader can insert notes or questions wherever desired and have 
the computer print cut any section at will. 

FHESS allows a student or professor to assemble such a text 
structure for himself or others; subsequent readers in turn can 
progressively enrich it by using it repeatedly, adding new items, 
rearranging cld ones, and answering questions or disputing 
answers left embedded by their colleagues. 


B..3_THE0RY of hypertext in e duc ation 


A hypertext system is intended to make the learning 
processes of the student richer and, in some sense, easier than 
is possible with traditional methods. This can be accomplished 
by taking advantage of the fact that most learning is in terms of 
conceptual associations: the association of new thoughts with 
various old ones. As Vannevar Bush said in his visionary essay 
on mechanical aids to scholarship 2 ': 

The human mind . . . operates by association. With one 
item in its grasp, it snaps instantly to the next that 
is suggested by the association of thoughts, in 
accordance with some intricate web of trails carried by 
the cells of the brain.... [The] intricacy of trails, 
the detail of mental pictures, is awe-inspiring beyond 
all else in nature. 

A hypertext system provides a framework in which teaching and 
learning can take advantage of these associative thought 
processes. At the same time it provides a means to record the 
thoughts of the student or researcher -- comments, questions, and 
new associative trails — thereby enriching not only the 
individual's learning experience but the experiences of later 
students or researchers as well. 

Conventional Computer Assisted Instruction (CAI) is quite 
different from a hypertext system. Computer assisted 
instructional schemes are rooted in the notion that if a body of 
knowledge can be broken 1 into small pieces and presented piece by 
piece to a student, then a proper use of machines, and 
particularly of computers, is to set them to work flashing such 
items to students in some pre-arranged sequence. The emphasis in 
such systems is to form a static, reproducible organization of 


24 Vannevar Bush, "As We May Think," Atlantic Monthly 176:1 (July, 
1945 ), p. 103. 
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known materials, with the goal of replacing or reducing the role 
of the teacher. 

Computers have proven to be an excellent means for attaining 
skills which require drill and practice or for teaching material 
conducive to prearranged instructional sequences. However, their 
application to this type of education has received a 

disproportionate amount of attention. Consider the college 
seminar studying literary movements in 19th-century Britain or 
the scholar assembling data from various sources concerning the 
sonnet. Typical computer assisted instruction is of little or no 
use in such situations: 

1) An individual doing research will certainly desire 
to develop original thoughts along new lines; CAI 
could only have value where the research is already 
done . 

2) The researcher will wish to record and develop 

thoughts and reactions to the material during 
browsing; CAI limits responses to predictable 
patterns . 

It may seem that we are mixing apples and oranges — research and 
instruction. But these features of a hypertext system that make 

it useful for research also make it useful for instruction on a 

level that cannot be handled with traditional CAI methods. 

The feature that is perhaps most significant in separating 
hypertext from conventional CAI is that the hypertext can be 
enhanced by the addition of text and/or links by a subsequent 
reader/student as well as by the original creator/teacher. This 
makes the interaction between student and material more dynamic 
and allows greater flexibility than in standard CAI applications. 
In addition, important and educationally beneficial 
student-student, student-instructor, and instructor-instructor 
interactions are possible, as discussed below. 



B . 4 EXXON 


A first attempt to apply this theory Oj. associative learning 
with hypertexts was carried out at Brown from tiie summer of 1973 
to the end of the spring semester in 1974 in an experiment funded 
by the Exxon Education Foundation. An interdisciplinary 
university-level course on energy was chosen as the vehicle. 
Twelve students participated in the experiment by doing all of 
their course reading and seme of their paper writing using FRESS. 


- 40 - 



Background of PRESS 


Plato a nd NLS 


The results cf the experiment were i nconcl usi ve. It was 
determined that the PRESS hypertext system was useable, 
effective, and enjoyable in an educational environment 
encompassing students with diverse backgrounds. However, the 
evidence from this single experiment was not sufficient to prove 
whether or not the hypertext helped the students to learn more or 
better. Also, while the students passively explored most of the 
instructor-created hypertext, there was insufficient incentive 
for them to enlarge it. 


B. 5 PLAT O A ND NLS 


Only two computer-based information systems have facilities 
comparable to those of FRESS. The PLATO IV system, developed by 
Donald Bitzer and others at the University of Illinois 25 , is the 
most sophisticated CAI system available. It allows a significant 
amount of user control in the choice of lesson and in user input 
in problem solving. For example, the user may choose a 
particular, predefined mathematical function and can then specify 
values of the independent variables — PLATO will draw the curve; 
or a user may "set up" a chemistry experiment -- PLATO will 
provide the data. The emphasis in PLATO is still typical of CAI: 
fact and skill acquisition by the method of drill and practice, 
with the computer system as an accommodating, sophisticated 
tutor. As with all CAI systems, there is a major distinction, 
not found in FRESS, in facilities available to teacher/authors 
and student/readers. PLATO has recently added a question/comment 
facility that serves as a type of community bulletin board. 
However, it does not provide the ability to connect these notes 
to sections of lesson material; thus the user can read lessons or 
read the bulletin board, but cannot mix the two functions and 
cannot determine if there is a comment on a lesson from looking 
at the lesson alone. Other sorts of links are just not possible 
at all. The "bulletin board" appears to be used mostly for 
notices of meetings, global discussions regarding preparation of 
new lessons, etc. The facility is thus much less generally 
useful than the FRESS g uestion/ comm ent facility. 

The sophisticated , NLS text processing system, developed by 
Douglas Engelbart at the Stanford Research Institute 26 has 
facilities comparable to those of FRESS. It combines a text 


25 Allen L. Hammond, "Computer Assisted Instruction: Two Major 
Demonstrations", Science 176: 1110-1111, June 9, 1972. 

26 Douglas Engelbart and William English, "A Research Center for 
Augmenting Human Intellect," Proceedings 1 968 FJCC, Vol . 33, Part 
I, AFI2S Press, Hontvale, N. J., pp. 395-410. 
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editor together with browsing and linking 
the text is entered in a "hierarchical" 
decimally numbered outline form. Since 
suppressed from view, this presents only 
dealing with text not structured in 
difference between FRESS and NLS is that 


capabilities. 

structure, a s 
the numbering 
minor difficult 
this manner. The 
NI.S has been desig 


be used by computer scientists and other technical sta 
document preparation and (tele) conferencing in their own 
It has not been used by non- tech nica 1 people in an eauca 
setting. In summary, to our knowledge only FRESS has bee 


in innovative experiments 
types of skills associated 
acquisition. 


that involve critical judgment a 
with scholarship rather than wit 


In NLS 
ort of 
can be 
ies in 
major 
ned to 
f f for 
work, 
tion al 
n used 
n d the 
h fact 


Statistics 


Evaluation data 


Q_S1A1IS1ICS 


(Section 1A used the computer-based hypertext; Section IE is 
the control section. Sections 2 and 3 were taught by other 


instructors, 
inf orma tion. ) 

See 

Section 3.3 

of 

this 

report 

for further 

Section 

1A 

11 

2 


3 

Total 

A 

3 

3 

4 


4 

1 4 

B 

4 

1 

10 


4 

19 

C 

1 

0 

1 


0 

2 

S 

4 

2 

7 


4 

17 

NC 

0 

1 

0 


1 

2 

I 

0 

1 

0 


3 

4 

Total 

12 

8 

22 


16 

58 

C . 2 EVALUATION 

DATA 






The following 

table lists. 

for 

each 

section 

, the score on 


each student's analysis of the poem before the course began, the 
score on that student's analysis after the course was finished, 
and the difference between these two scores. For a discussion of 
the results and an additional table see Section 4. 2. 3. 3. 




1A 



IB 




2 




3 


7 

— 

10 

+3 

7 - 

7 

0 

5 

— 

7 

+ 2 

7 

- 

5 

-2 

6 

- 

9 

+ 3 

6 - 

8 

+ 2 

7 

- 

8 

+ 1 

8 

- 

6 

-2 

5 

- 

7 

+ 2 

7 - 

8 

♦ 1 

7 

- 

6 

- 1 

9 

- 

6 

-3 

7 

- 

9 

+ 2 

5 - 

6 

+ 1 

10 

- 

6 

-4 

5 

- 

5 

0 

9 

- 

6 

-3 

4 - 

5 

+ 1 

9 

- 

9 

0 

9 

- 

7 

-2 

6 

- 

6 

0 


» 


9 

- 

10 

+ 1 

7 

- 

8 

+ 1 

5 

- 

8 

♦ 3 




5 

- 

7 

4 2 

6 

- 

6 

0 

7 

- 

8 

+ i 




5 

- 

7 

+ 2 

7 

- 

6 

-1 

7 

- 

9 

f 1 




7 

- 

7 

0 

5 

- 

8 

+ 3 

8 

- 

9 

+ 2 




5 

- 

6 

+ 1 

7 

- 

7 

0 

7 

- 

7 

0 




8 

- 

7 

- 1 

8 

- 

6 

-2 

8 

- 

8 

0 




7 

- 

8 

+ 1 

7 

- 

9 

+ 2 








7 

- 

7 

0 












6 

- 

7 

♦ 1 












4 

- 

7 

+ 3 
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Usage Information 


C^3 USAGE INFORMATION 


Key: 
OND = 

Total hours during 

Oc tobe r , 

No vember , 

and December for 

/ W K — 

each student 

The weekly average 

usage of 

the system 

during October, 

JAN = 

November, 
Number of 

and December (10.2 weeks), 
hours used during January 

Total = 
PPS = 

Sum of OND 
Number of 

and JAN 

pages (equivalent) 

of written work for each 


student - 
OND 

/WK 

JAN 

Total 

PPS 

1 

50.01 

4.90 

13. 30 

63. 31 

57.16 

2 

59.47 

5.83 

18.33 

77.80 

83.4S 

3 

56.80 

5. 57 

2.32 

59. 12 

80.40 

4 

46.74 

4.58 

12.18 

58.92 

82.60 

5 

74.15 

7. 26 

28. 27 

102. 42 

92.0 4 

6 

56.00 

5.49 

.03 

56.03 

78.48 

7 

54.08 

5. 30 

17.98 

72.06 

59 .40 

6 

57.77 

5.66 

22.48 

80.25 

112.16 

9 

73.73 

7.22 

17.83 

91.56 

63.20 

10 

67.98 

6.66 

11.90 

79.88 

73. 32 

1 1 

48.49 

4.75 

13.77 

62.26 

91 .04 

12 

56.57 

5. 55 

18.47 

75.04 

137.00 

Av er age 

58.48 

5.73 

14. 73 

73. 22 

84.19 
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D.E^EENDI^URgS 27 


IA : Preparation 

BROWN 

NEH 

Salaries and Benefits 
Computer funds 
M iscellaneous 

2, 735 

33,810 

3,989 

4,604 

IB: Data base collection 



Typist salaries and benefits 
Computer funding 


1 , 736 
2,476 

II: Student Use 



Computer funds (934 hours) 

3 ,804 

2,385 

III: Film 



Salaries, supplies, film 
developing 


10,200 

IV : Overhead 


19,636 

Tot al 

6, 539 

78,836 


27 These figures are estimates. 
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